Image processing method and apparatus for head-mounted display device as well as electronic device

ABSTRACT

Embodiments of the present disclosure disclose an image processing method for a head-mounted display device, an apparatus and an electronic device. The method includes: acquiring a to-be-processed image and eyeball-tracking information; determining a first region of interest of an eyeball on the to-be-processed image according to the eyeball-tracking information; assigning a weight value to each pixel point in the to-be-processed image, wherein weight values of pixel points located in the first region of interest are higher than weight values of pixel points located in other regions which are regions other than the first region of interest; adjusting a pixel value of each pixel point according to the weight value corresponding thereto so as to obtain a processed image.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of InternationalApplication No. PCT/CN2021/118172, filed Sep. 14, 2021, which claimspriority to the Chinese patent application Number 202010974865.3entitled “IMAGE PROCESSING METHOD AND APPARATUS FOR HEAD-MOUNTED DISPLAYDEVICE AS WELL AS ELECTRONIC DEVICE” and filed with the China PatentOffice on Sep. 16, 2020, both of which are hereby incorporated byreference in their entireties.

TECHNICAL FIELD

Embodiments of the present disclosure relates to the field of videoprocessing technology, and specifically, the embodiments of the presentdisclosure relate to an image processing method for a head-mounteddisplay device, an image apparatus for the head-mounted display device,and an electronic device.

BACKGROUND

Virtual Reality (VR) is an emerging high-tech in recent years. With thebooming development of virtual reality industry, there is an increasingdemand for interaction between virtual and reality during user's use.

At present, video coding technology is widely used in the field of VR.When a user broadcasts live VR videos, watches VR videos and experiencesVR games through a VR head-mounted display device at a PC (PersonalComputer) terminal, it is necessary to encode and compress the videocontent rendered by the PC through the video coding technology, andtransmit the encoded and compressed video file to the head-mounteddisplay device through a network, so that the head-mounted displaydevice can present it to the user. To facilitate transmission of thevideo, it is common to simply partition the images to be rendered andreduce the resolution of the area of non-concern to reduce video size,thereby increasing transmission speed and shortening display delay ofthe head-mounted display device. When the video is processed in thisway, the area of concern and the area of non-concern of the image areabruptly separated and generate a large contrast therebetween, therebyinfluencing display effect of the video. In addition, it is likely tocause fusion of pixel points due to the reduced resolution of the areaof non-concerned area, resulting in a large number of bright spots andthus influencing the visual experience of the user.

Therefore, it is necessary to provide a new image processing method fora head-mounted display device so as to reduce the video size whilemaintaining its quality.

SUMMARY

An objective of embodiments of the present disclosure is to provide animage process scheme which can reduce the video size while maintainingits quality.

According to a first aspect of the embodiments of the presentdisclosure, an image processing method for a head-mounted display deviceis provided, including:

acquiring a to-be-processed image and eyeball-tracking information;

determining a first region of interest of an eyeball on theto-be-processed image according to the eyeball-tracking information;

assigning a weight value to each pixel point in the to-be-processedimage, wherein weight values of pixel points located in the first regionof interest are higher than weight values of pixel points located inother regions which are regions other than the first region of interest;

adjusting a pixel value of each pixel point according to the weightvalue corresponding thereto so as to obtain a processed image.

Optionally, the eyeball-tracking information includes a center point anda radius of a fixation area of the eyeball on the to-be-processed image,and the maximum movement angle of the eyeball on the to-be-processedimage within a reception interval; determining a first region ofinterest of an eyeball on the to-be-processed image according to theeyeball-tracking information includes:

determining the maximum displacement of the eyeball on theto-be-processed image within the reception interval according to themaximum movement angle;

determining the first region of interest according to the center pointand the maximum displacement of the fixation area if the maximumdisplacement is larger than or equal to the radius of the fixation area;

determining the fixation area as the first region of interest if themaximum displacement is smaller than the radius of the fixation area.

Optionally, the method further includes determining the maximum movementangle of the eyeball on the to-be-processed image, including:

acquiring an actual movement angle of the eyeball on the to-be-processedimage within the reception interval;

taking a field of view of the head-mounted display device as the maximummovement angle of the eyeball on the to-be-processed image within thereception interval if the actual movement angle is larger than or equalto the field of view of the head-mounted display device;

taking the actual movement angle as the maximum movement angle of theeyeball on the to-be-processed image within the reception interval ifthe actual movement angle is smaller than the field of view of thehead-mounted display device.

Optionally, assigning a weight value to each pixel point in theto-be-processed image includes:

assigning a first weight value to each pixel point in theto-be-processed image, wherein the first weight value corresponding tothe pixel point tends to decrease from a central axis of theto-be-processed image to edges on both sides of the same;

assigning a second weight value to each pixel point in the first regionof interest, wherein the second weight value corresponding to the pixelpoint tends to decrease from a center to edge of the first region ofinterest;

performing weighted average calculation on the first weight value andthe second weight value corresponding thereto in the first region ofinterest, so as to determine a third weight value of each pixel point inthe first region of interest.

Optionally, assigning a weight value to each pixel point in theto-be-processed image includes:

partitioning the to-be-processed image so as to obtain a plurality offirst image regions;

assigning a fourth weight value to each of the first image regions,wherein the fourth weight value corresponding to the first image regionstends to decrease from a central axis of the to-be-processed image toedges on both sides of the same;

partitioning the first region of interest so as to obtain a plurality ofsecond image regions;

assigning a fifth weight value to each of the second image regions,wherein the fifth weight value corresponding to the second image regionstends to decrease from a center point of the first region of interest toan edge of the same;

determining a weight value of each pixel point in the to-be-processedimage according to the fourth weight value corresponding to the firstimage region where the pixel point is located and the fifth weight valuecorresponding to the second image region where the pixel point islocated.

Optionally, adjusting a pixel value of each pixel point according to theweight value corresponding thereto so as to obtain a processed imageincludes:

determining an image information Qstep corresponding to each pixel pointin the to-be-processed image according to a pre-established mappingcorrespondence between the weight value and the image information Qstep;

adjusting the pixel value of each pixel point according to the imageinformation Qstep corresponding thereto so as to obtain a processedimage.

Optionally, the method further includes:

performing video coding on a plurality of frames of the processed imagearranged in sequence so as to obtain a target video.

Optionally, the method further includes:

performing convolution operation on the processed image so as to obtaina convolution operation result;

determining a second region of interest of the processed image accordingto the convolution operation result;

based on the second region of interest, performing video coding on aplurality of frames of the processed image arranged in sequence with apredetermined region-of-interest coding algorithm so as to obtain atarget video.

According to a second aspect of the embodiment of the presentdisclosure, an image processing apparatus for a head-mounted displaydevice is provided, the apparatus includes:

an acquisition module configured to acquire a to-be-processed image andeyeball-tracking information;

a first region-of-interest determining module configured to determine afirst region of interest of an eyeball on the to-be-processed imageaccording to the eyeball-tracking information;

a weight value determining module configured to assign a weight value toeach pixel point in the to-be-processed image, wherein weight values ofpixel points located in the first region of interest are higher thanweight values of pixel points located in other regions which are regionsother than the first region of interest;

an image processing module configured to adjust a pixel value of eachpixel point according to the weight value corresponding thereto so as toobtain a processed image;

or,

the apparatus includes a processor and a memory, the memory storingcomputer instructions that, when run by the processor, execute themethod of any one of the first aspect of the embodiment of the presentdisclosure.

According to a third aspect of the embodiment of the present disclosure,an electronic device is provided, including a head-mounted displaydevice and an image processing apparatus for the head-mounted displaydevice of the second aspect of the embodiment of the present disclosure.

According to the embodiment of the present disclosure, the first regionof interest of the eyeball on the to-be-processed image is predictedaccording to the eyeball-tracking information, each pixel point in theto-be-processed image is assigned a weight value, and the pixel value ofthe pixel point is adjusted according to the weight value correspondingto the pixel point, so that it is possible to reduce the complexity ofimage processing while ensuring the image quality.

According to the embodiment of the present disclosure, the processedimage can maintain the original state of the image in the first regionof interest, and enable the chroma, contrast, and the like of the imageto be naturally transitioned according to the weight values of pixelpoints in regions other than the first region of interest, thereforemaking the display effect of the image better.

Other features and advantages of the present disclosure will becomeapparent from the following detailed description of exemplaryembodiments of the present disclosure with reference to the accompanyingdrawings.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to illustrate the technical solutions of the embodiments of thepresent disclosure more clearly, the following is a brief introductionof the attached drawings required in the embodiment. It should beunderstood that the following drawings show only some embodiments of thedisclosure and should not be considered a limitation of the scope. Forthose generally skilled in the art, other relevant attached drawings mayalso be obtained based on these attached drawings without any creativeeffort.

FIG. 1 is a schematic diagram of a hardware configuration of anelectronic device that may be used to implement an embodiment of thepresent disclosure;

FIG. 2 is a schematic flowchart of an image processing method for ahead-mounted display device according to an embodiment of the presentdisclosure;

FIG. 3 is a schematic diagram of the maximum displacement of an eyeballon a to-be-processed image according to an embodiment of the presentdisclosure;

FIG. 4 is a schematic diagram of a first region of interest according toan embodiment of the present disclosure;

FIG. 5 is a schematic diagram of a first region of interest according toan embodiment of the present disclosure;

FIG. 6 is a schematic diagram of partitioning a to-be-processed imageaccording to an embodiment of the present disclosure;

FIG. 7 is a structural block diagram of an image processing apparatusfor a head-mounted display device according to an embodiment of thepresent disclosure;

FIG. 8 is a structural block diagram of an image processing apparatusfor a head-mounted display device according to an embodiment of thepresent disclosure.

DETAILED DESCRIPTION

Various exemplary embodiments of the present disclosure will now bedescribed in detail with reference to the accompanying drawings. Itshould be noted that the relative arrangement, numerical expressions andnumerical values of the components and steps set forth in these examplesdo not limit the scope of the disclosure unless otherwise specified. Thefollowing description of at least one exemplary embodiment is in factmerely illustrative and is in no way intended as a limitation to thepresent disclosure and its application or use.

Techniques, methods, and apparatus known to those of ordinary skill inthe relevant art may not be discussed in detail but where appropriate,the techniques, methods, and apparatus should be considered as part ofthe description.

Among all the examples shown and discussed herein, any specific valueshould be construed as merely illustrative and not as a limitation.Thus, other examples of exemplary embodiments may have different values.

It should be noted that similar reference numerals and letters denotesimilar items in the accompanying drawings, and therefore, once an itemis defined in a drawing, and there is no need for further discussion inthe subsequent accompanying drawings.

<Hardware Configuration>

FIG. 1 shows a block diagram of a hardware configuration of anelectronic device 100 provided by an embodiment of the presentdisclosure.

As shown in FIG. 1 , the electronic device 100 includes a head-mounteddisplay device 1000 and a PC end 2000. The head-mounted display device1000 is connected in communication with the PC end 2000.

In the embodiment of the present disclosure, the head-mounted displaydevice 1000 may be applied to scenarios such as broadcasting live VRvideos, watching VR videos, and experiencing VR games. The head-mounteddisplay device 1000 is connected to the PC end 2000 for acquiring atarget video from the PC end 2000 and presenting it to a user.

The PC end 2000 is used to process a to-be-processed image according topose data of the head-mounted display device 1000, perform video codingon the sequentially arranged multi-frame processed images to obtain thetarget video, and transmit the target video to the head-mounted displaydevice 1000. The target video may be, for example, a VR panoramic video,a VR game video, or the like.

The head-mounted display device 1000 may be, for example, a VR (VirtualReality) device, an AR (Augmented Reality) device, an MR (Mixed Reality)device, or the like.

In one embodiment, the head-mounted display device 1000 may be shown asin FIG. 1 , including a processor 1100, a memory 1200, an interfaceapparatus 1300, a communication apparatus 1400, a display apparatus1500, an input apparatus 1600, an audio apparatus 1700, and an inertialmeasurement unit 1800, a camera 1900, etc.

Wherein, the processor 1100 may be, for example, a central processingunit CPU, a microprocessor MCU, etc. The memory 1200 includes, forexample, a ROM (Read Only Memory), a RAM (Random Access Memory), anonvolatile memory such as a hard disk, etc. The interface apparatus1300 includes, for example, a serial bus interface (including a USBinterface), a parallel bus interface, a high-definition multimediainterface (HDMI interface), etc. The communication apparatus 1400 iscapable of, for example, wired or wireless communication. The displayapparatus 1500 is, for example, a liquid crystal display, an LEDdisplay, a touch display, or the like. The input apparatus 1600includes, for example, a touch screen, a keyboard, a somatosensoryinput, etc. The audio apparatus 1700 may be configured to input/outputvoice information. The inertial measurement unit 1800 may be configuredto measure changes in the pose of the head-mounted display device 1000.The camera 1900 may be configured to acquire image information.

Although a plurality of apparatuses are shown for the head-mounteddisplay device 1000 in FIG. 1 , the present disclosure may only involvesome of the apparatuses, for example, the head-mounted display device1000 only involves the processor 1100 and the memory 1200.

The PC end 2000 may be a computer with an image processing function. Inone embodiment, the PC end 2000 may be as shown in FIG. 1 , including aprocessor 2100, a memory 2200, an interface apparatus 2300, acommunication apparatus 2400, a display apparatus 2500, and an inputapparatus 2600. The processor 2100 may be a desktop processor, a mobileprocessor, etc. that meets performance requirements and is not limitedherein. The memory 2200 includes, for example, a ROM (Read Only Memory),a RAM (Random Access Memory), a nonvolatile memory such as a hard disk,etc. The interface apparatus 2300 includes, for example, various businterfaces, such as a serial bus interface (including a USB interface),a parallel bus interface, a high-definition multimedia interface (HDMIinterface), etc. The communication apparatus 2400 is capable of, forexample, wired or wireless communication. The display apparatus 2500 is,for example, a liquid crystal display, an LED display, a touch display,or the like. The input apparatus 2600 may include, for example, a touchscreen, a keyboard, etc. In another embodiment, the PC end 2000 mayfurther include a speaker, a microphone and the like, which is notlimited herein. Although a plurality of apparatuses of the PC end 2000are shown in FIG. 1 , the present disclosure may only involve some ofthe apparatuses. For example, the PC end 2000 only involves the memory2200 and the processor 2100.

In one embodiment, the memory 2200 of the PC end 2000 is configured tostore instructions which are configured to control the processor 2100 toexecute corresponding steps, so as to provide relevant support for theimage processing method for the head-mounted display device of thepresent embodiment.

It should be understood that although FIG. 1 only shows one head-mounteddisplay device 1000 and one PC end 2000, it does not mean to limit theirrespective numbers. The electronic device 100 may include a plurality ofhead-mounted display devices 1000 and a plurality of PC ends 2000.

In one embodiment, the head-mounted display device 1000 may also beconnected in communication with a server. The server may have a hardwareconfiguration similar to that of the PC end 2000. The server may beconnected in communication with the head-mounted display device 1000.The memory of the server is configured to store instructions which areconfigured to control the processor of the server to executecorresponding steps, so as to provide relevant support for the imageprocessing method for a head-mounted display device of the presentembodiment.

In the above description, the skilled person can design instructionsaccording to the scheme provided by the present disclosure. Howinstructions control the operation of a processor is well known in theart and thus will not be described in detail herein.

The electronic device shown in FIG. 1 is merely illustrative and is inno way intended to limit the disclosure, its application, or usesthereof.

<Embodiment of Method>

Referring to FIG. 2 , an image processing method for a head-mounteddisplay device according to an embodiment of the present disclosure isdescribed. The method relates to a PC end or a server which may beconnected with the head-mounted display device, is used for processingthe to-be-processed image according to the pose data of the head-mounteddisplay device, performs a video coding on the sequentially arrangedmulti-frame processed images to obtain a target video, and transmits thetarget video to the head-mounted display device for to presenting it toa user through the head-mounted display device. The PC end may be a PCend 2000 as shown in FIG. 1 , and the head-mounted display device may bea head-mounted display device 1000 as shown in FIG. 1 .

The image processing method for a head-mounted display device includesthe following steps S201-S204.

S201: acquiring a to-be-processed image and eyeball-trackinginformation.

The to-be-processed image may be obtained from the target video fordisplay on the head-mounted display device. The target video may begenerated according to the running program of the head-mounted displaydevice. The target video may be, for example, a VR panoramic video, a VRgame video, or the like. The target video includes a plurality of framesof to-be-processed images arranged in sequence, and is obtained byprocessing each frame of to-be-processed image so as to carry out videocoding on the plurality of frames of processed images arranged insequence.

The eyeball-tracking information may be obtained from the head-mounteddisplay device. More specifically, the eyeball-tracking information maybe collected by an eyeball-tracking module set on the head-mounteddisplay device. According to the eyeball-tracking information, a firstregion of interest of the eyeball on the to-be-processed image may bedetermined. By processing the to-be-processed image based on the firstregion of interest, it is possible to reduce the complexity of the imagewhile ensuring the quality of the image.

After acquiring the to-be-processed image and the eyeball-trackinginformation, it proceeds to step S202.

S202: determining the first region of interest of the eyeball on theto-be-processed image according to the eyeball-tracking information.

In one embodiment, the eyeball-tracking information may include a centerpoint and a radius of a fixation area of the eyeball on theto-be-processed image, and a maximum movement angle of the eyeball onthe to-be-processed image in a reception interval. In this embodiment,the step of determining the first region of interest of the eyeball onthe to-be-processed image according to the eyeball-tracking informationmay further include S301-S303.

S301: determining the maximum displacement of the eyeball on theto-be-processed image during a reception interval according to themaximum movement angle.

The reception interval may reflect the time delay between theacquisition time of the eyeball-tracking information and the collectiontime of the eyeball-tracking information. The reception interval may beset according to engineering experience or simulation test experience.There is a certain time delay between the time when the PC end or theserver acquires the eyeball-tracking information and the time when theeyeball-tracking information is collected. According to the center pointand the radius of the fixation area of the eyeball on theto-be-processed image and the maximum movement angle of the eyeball onthe to-be-processed image in the reception interval, the maximumdisplacement of the main eyeball on the to-be-processed image may bedetermined, and the first region of interest may be further predicted.

In one embodiment, the image processing method for the head-mounteddisplay device further includes a step of determining an object movementangle, which may include S401-S403.

S401: acquiring the actual movement angle of the eyeball on theto-be-processed image in the reception interval.

In a more specific example, the actual movement angle of the eyeball onthe to-be-processed image in the reception interval may be collecteddirectly through the eyeball-tracking module set on the head-mounteddisplay device.

In another more specific example, the actual movement angle of theeyeball on the to-be-processed image in the reception interval may bedetermined according to an eyeball movement speed and the receptioninterval. Specifically, the maximum movement angle of the eyeball on theto-be-processed image in the reception interval is calculated based onthe following formula (1).

D=V*S  (1)

Wherein, D is the maximum movement angle of the eyeball on theto-be-processed image within the reception interval, V is the eyeballmovement speed, and S is the reception interval. The reception intervalS and the eyeball movement speed V may be set based on practicalexperience. According to statistics, the eyeball may be focusedeffectively at a speed of 2 degrees/sec. Based on this, the eyeballmovement speed may be, for example, 2 degrees/sec.

S402: taking a field of view of the head-mounted display device as themaximum movement angle of the eyeball on the to-be-processed imagewithin the reception interval under a condition that an actual movementangle is greater than or equal to the field of view of the head-mounteddisplay device.

S403: taking the actual movement angle as the maximum movement angle ofthe eyeball on the to-be-processed image within the reception intervalwhen the actual movement angle is smaller than the field of view of thehead-mounted display device.

In this embodiment, the PC end or the server may acquire deviceparameters of the head-mounted display device in advance, which includethe field of view of the head-mounted display device and a 3D projectionradius of the head-mounted display device. The field of view of thehead-mounted display device may reflect a sight range that the user canview through the head-mounted display device. The field of view may be ahorizontal field of view. If the actual movement angle is greater thanor equal to the field of view of the head-mounted display device, itindicates the actual movement angle of the eyeball is beyond the sightrange of the head-mounted display device. At this time, the field ofview of the head-mounted display device is taken as the maximum movementangle of the eyeball on the to-be-processed image within the receptioninterval. If the actual movement angle is less than the field of view ofthe head-mounted display device, it indicates that the actual movementangle of the eyeball is within the sight range of the head-mounteddisplay device. At this time, the actual movement angle is the maximummovement angle of the eyeball on the to-be-processed image within thereception interval.

After the maximum movement angle of the eyeball on the to-be-processedimage within the reception interval is determined, the maximumdisplacement of the eyeball on the to-be-processed image within thereception interval is determined according to the maximum movementangle.

In a more specific example, the maximum displacement of the eyeball onthe to-be-processed image within the reception interval is determinedaccording to the maximum movement angle and the three-dimensionalprojection radius of the head-mounted display device. Specifically, themaximum displacement of the eyeball on the to-be-processed image withinthe reception interval is calculated based on the following formula (2).

L=sin D*R  (2)

Wherein, L is the maximum displacement of the eyeball on theto-be-processed image within the reception interval; D is the maximummovement angle of the eyeball on the to-be-processed image within thereception interval; R is the three-dimensional projection radius of thehead-mounted display device.

The principle of calculating the maximum displacement L of the eyeballon the to-be-processed image in the reception interval will be describedwith reference to FIG. 3 . As shown in FIG. 3 , the point A is theposition of the camera of the head-mounted display device (i.e., theposition of the eyeball), the point B is the center point of thefixation area of the eyeball on the to-be-processed image, the linesegment AC is the three-dimensional projection radius R of thehead-mounted display device, and the line segment BC is the maximumdisplacement of the eyeball on the to-be-processed image within thereception interval. Based on this, the maximum displacement L of theeyeball on the to-be-processed image within the reception interval maybe calculated according to the formula (2). The maximum displacement Lof the eyeball on the to-be-processed image within the receptioninterval calculated according to the formula (2) is an approximatevalue. According to the embodiment of the present disclosure, themaximum displacement obtained based on the formula (2) may reflect thereal movement displacement of the eyeball within the reception interval.According to the embodiment of the present disclosure, the algorithm maybe simplified by determining the maximum displacement of the eyeball onthe to-be-processed image within the reception interval based onEquation (2). According to an embodiment of the present disclosure, itis possible to simplify the algorithm by determining the maximumdisplacement of the eyeball on the to-be-processed image within thereception interval based on the formula (2).

S302: in the case where the maximum displacement is greater than orequal to the radius of the fixation area, determining the first regionof interest according to the center point of the fixation area and themaximum displacement.

In this embodiment, in the case where the maximum displacement isgreater than or equal to the radius of the fixation area, it indicatesthat when the PC end or the server acquires the eyeball-trackinginformation, the maximum possible displacement of the eyeball on theto-be-processed image exceeds the determined fixation area. Therefore,the region of interest of the eyeball on the to-be-processed image maybe re-predicted according to the maximum displacement of the eyeball onthe to-be-processed image within the reception interval, that is,obtaining the first region of interest. The fixation area is expanded bythe first region of interest determined by the embodiment of the presentdisclosure, which may avoid the influence of time delay between theacquisition time of the eyeball-tracking information and the collectiontime of the eyeball-tracking information, and improve the accuracy ofpredicting the region of interest, thereby facilitating the subsequentimage processing.

The first region of interest may include any regular or irregular regionof the fixation area, that is, the extent of the first region ofinterest is greater than the extent of the fixation area. The firstregion of interest may be a circular region or an elliptical region.

In the case where the first region of interest is an elliptical region,see FIG. 4 , a line segment is made along the moving direction of theeyeball with the center point of the fixation area as the startingpoint, the length of the line segment is the maximum displacement L ofthe eyeball on the to-be-processed image within the reception interval,and two vertexes of the line segment are taken as two foci of theellipse to form the elliptical region being the first region ofinterest. The size of the elliptical area may depend on the size of thefixation area. For example, the elliptical area includes at least thefixation area.

S303, in the case where the maximum displacement is smaller than theradius of the fixation area, determining the fixation area as the firstregion of interest.

In this embodiment, when the maximum displacement is less than theradius of the fixation area, it indicates that when the eyeball-trackinginformation is acquired by the PC end or the server, the maximumpossible displacement of the eyeball on the to-be-processed image iswithin the fixation area. In this regard, fixation area is used as thefirst region of interest.

The fixation area may be any regular or irregular region, that is, thefirst region of interest may be any regular or irregular region. Thefirst region of interest may be a circular region or an ellipticalregion. In the case where the first region of interest is a circularregion, referring to FIG. 5 , the first region of interest is a circularregion with a center point of the fixation area as a dot and a radius ofr.

After determining the first region of interest of the eyeball on theto-be-processed image, it proceeds to step S203.

S203: assigning a weight value to each pixel point in theto-be-processed image, wherein weight values of the pixel points locatedin the first region of interest are higher than weight values of thepixel points located in other regions which are regions other than thefirst region of interest.

In this embodiment, a higher weight value is assigned to the pixelpoints in the first region of interest, and a lower weight value isassigned to other regions, and the pixel values of the pixel pointslocated in different regions are adjusted to different degrees accordingto the weight values corresponding to the pixel points, which reducesthe image information of the pixel points in the regions with lowerweight values while retaining the image information of the first regionof interest, so that it is possible to reduce the complexity of imageprocessing so as to improve the processing speed while ensuring theimage quality.

In one embodiment, each pixel point in the to-be-processed image may beassigned a weight value according to the position of the pixel point. Inanother embodiment, the to-be-processed image may also be subjected topartition, and weight values are assigned to the partitioned imageregions, wherein the weight values of all pixel points located in theimage region are the same.

A process of assigning a weight value to each pixel point in theto-be-processed image according to the position of the pixel point willbe described below. The process may include S501-S503.

S501: assigning a first weight value to each pixel point in theto-be-processed image, wherein the first weight value corresponding tothe pixel point tends to decrease from the central axis to the edges onboth sides.

In this embodiment, the central axis may be a transverse central axis.The first weight value corresponding to the pixel point tends todecrease from the central axis to the edges on both sides, that is, thefirst weight value corresponding to the pixel point decreases graduallyfrom the central axis to the edges on both sides. Distances of the pixelpoints from the center axis are different, and the degree of distortionof the image is also different, so that the first weight value may beassigned according to distances of the pixel points in theto-be-processed image from the center axis.

S502: assigning a second weight value to each pixel point in the firstregion of interest, where the second weight value corresponding to thepixel point tends to decrease from the center to the edge of the firstregion of interest.

In this embodiment, the second weight value is assigned according to thedistance of the pixel point from the center point of the first region ofinterest. The second weight value corresponding to the pixel point tendsto decrease from the center to the edge of the first region of interest,that is, the closer the pixel point is to the center point of the firstregion of interest, that is, the greater the degree of interest is, ahigher second weight value is assigned. The farther the pixel point isfrom the center point of the first region of interest, that is, thesmaller the degree of interest is, the lower second weight value isassigned.

According to the embodiment of the invention, the second weight value isassigned according to the distance of the pixel point from the centerpoint of the first region of interest, and pixel value of each pixelpoint of the image is adjusted based on different weight values incombination with the subsequent steps so as to reduce the imageinformation of the pixel points in areas other than the first region ofinterest, which reasonably reduces the chroma and contrast of a localarea of the to-be-processed image, thereby ensuring the image quality ofthe first region of interest while reducing the complexity of imageprocessing. According to the embodiment of the present disclosure, thesecond weight value corresponding to the pixel point gradually decreasesfrom the center to the edge of the first region of interest, which canmake the image change smoother and improve the display effect of theimage.

S503: performing weighted average calculation on the first weight valueand the second weight value corresponding thereto in the first region ofinterest to determine a third weight value of each pixel point in thefirst region of interest.

In this embodiment, the pixel point located in the first region ofinterest has a first weight value and a second weight value, so that thethird weight value of the pixel point located in the first region ofinterest may be determined according to the first weight value and thesecond weight value.

According to the embodiment of the present disclosure, a weight value isassigned to each pixel point in the to-be-processed image, and the pixelvalue of each pixel point is adjusted differently according to differentweight values, so as to reduce the image information of the pixel pointsin regions other than the first region of interest and reserve moreimage information for the pixel points in the first region of interest,which reasonably reduces the chroma and contrast of a local area of theto-be-processed image, thereby ensuring the image quality of the firstregion of interest while reducing the complexity of image processing.

According to the embodiment of the present disclosure, the first weightvalue corresponding to the pixel point is gradually reduced from thecentral axis to the edges on both sides, and the second weight valuecorresponding to the pixel point is gradually reduced from the center ofthe first region of interest to the edge, which can make the imagechange smoother, and avoid abrupt transition between the region ofinterest and the region of non-interest, thereby improving the displayeffect of the image.

A process of assigning the weight values according to the partitionedimage regions will be described below. The process may includeS601-S605.

S601: partitioning the to-be-processed image to obtain a plurality offirst image regions.

In this embodiment, the to-be-processed image may be partitionedaccording to the field of view of the head-mounted display device, thatis, the to-be-processed image is partitioned into regions along theextension direction of the horizontal field of view. The first imageregion may be any regular or irregular region. For example, the firstimage region may be a strip region.

S602: assigning a fourth weight value to each first image region,wherein the fourth weight value corresponding to the first image regiondecreases from the central axis to the edges on both sides.

Each first image region specifically has a different fourth weightvalue. According to the weight value of the first image region, it ispossible to determine weight values of the pixel points in the firstimage region, wherein the weight values of each pixel point in the firstregion are the same.

S603: partitioning the first region of interest to obtain a plurality ofsecond image regions.

In this embodiment, the partition may be performed from the center pointto the edge of the first region of interest. The second image region maybe any regular or irregular region. For example, the second image regionmay be a circular region or an annular region.

S604: assigning a fifth weight value to each second image region,wherein the fifth weight value corresponding to the second image regiontends to decrease from the center point to the edge of the first regionof interest.

Each second image region specifically has a different fourth weightvalue. According to the weight value of the second image region, it ispossible to determine the weight values of the pixel points located inthe second image region, wherein the weight values of each pixel pointlocated in the second image region are the same.

S605: determining a weight value of each pixel in the to-be-processedimage according to the fourth weight value corresponding to the firstimage region where the pixel is located and the fifth weight valuecorresponding to the second image region where the pixel is located.

In this embodiment, the pixel point located in the first region ofinterest may be located in one of the first image regions or in one ofthe second image regions. Based on this, it is possible to determine theweight value of each pixel point in the to-be-processed image accordingto the fourth weight value corresponding to the first image region wherethe pixel point is located and the fifth weight value corresponding tothe second image region where the pixel point is located.

The process of assigning the weight value according to the partitionedimage regions will be described below with a more specific example inconjunction with the accompanying drawings.

Step 1: symmetrically partitioning the left part and the right part ofthe to-be-processed image along a transverse central axis of theto-be-processed image to obtain first image regions of 2n, wherein n isa positive integer and is more than or equal to 2. A weight value W_(1n)is assigned to each first image region, and the weight value of thefirst image region decreases progressively from the transverse centralaxis of the to-be-processed image to the left and right edges. Forexample, referring to FIG. 6 , the to-be-processed image is partitionedinto first image regions of 2*6, the weight values of the first imageregion to the left from the transverse central axis of theto-be-processed image are W₁₁, W₁₂, W₁₃, W₁₄, W₁₅, and W₁₆,respectively, and the weight values of the first image region to theright from the transverse central axis of the to-be-processed image areW₁₁, W₁₂, W₁₃, W₁₄, W₁₅ and W₁₆, respectively.

Step 2: circular partitioning is performed from the center point to theedge of the first region of interest to obtain second image regions ofk, wherein k is a positive integer and is more than or equal to 2. Thatis, the first region of interest is partitioned into a central circularregion and a plurality of outer annular regions. Each second imageregion is assigned a weight W_(2k), and the weight values of the secondimage region decreases progressively from the center point to the edgeof the first region of interest. Referring to FIG. 6 , the first regionof interest of the to-be-processed image is partitioned into four secondimage regions (a circular region in the center and three annular regionson the outer side), and the weight values of the second image regionfrom the center point to the edge of the first region of interest areW₂₁, W₂₂, W₂₃ and W₂₄, respectively.

Step 3: determining a weight value corresponding thereto in theto-be-processed image. Specifically, the weight value correspondingthereto is calculated based on the following formula (3).

W _((x,y)) =K ₁ *W _(1n) +K ₂ *W _(2k)  (3)

Wherein, W_((x,y)) is a weight value corresponding thereto in theto-be-processed image, W_(in) a weight value corresponding to the firstimage region, K₁ a weighting coefficient corresponding to the firstimage region, W₂K is a weight value corresponds to the second imageregion, and K₂ the weight value corresponding to the second imageregion.

In this step, when the pixel point is located in the first image region,the weighting coefficient corresponding to the first image region is 1,and the weighting coefficient corresponding to the second image regionis 0, that is, the weight value corresponding to this pixel point is theweight value corresponding to the first image regions where this pixelpoint is located. When the pixel is located in an overlapping region ofthe first image region and the second image region, that is, when thepixel point is located in the first region of interest, the weight valuecorresponding to the pixel point may be calculated according to theabove formula (3). For example, referring to FIG. 6 , the weight valuecorresponding to the pixel point (x₁,y₁) is W₁₃, and the weight valuecorresponding to the pixel point (x₂,y₂) is (K₁*W₁₁+K₂*W₂₃).

According to the embodiment of the present disclosure, theto-be-processed image is subjected to partition, a weight value isassigned to each image region, and the weight values of the pixel pointsmay be further determined according to the weight values of the imageregions, so that the pixel value of each pixel may be adjusted todifferent degrees according to the weight value of each pixel point incombination with the subsequent steps. According to the image processingmethod provided by the embodiment of the present disclosure, it ispossible to reduce the image information of the pixel points in regionsother than the first region of interest and reserve more imageinformation for the pixel points in the first region of interest so asto reasonably reduce the chroma and contrast of a local area of theto-be-processed image, thereby ensuring the image quality of the firstregion of interest while reducing the complexity of image processing.

According to the embodiment of the present disclosure, by partitioningthe to-be-processed image and assigning a weight value to each imageregion, pixel points located in the same image region may have the sameweight value, such that the pixel values of the pixel points located inthe same image region may be adjusted in the same way in combinationwith the subsequent steps, thereby further reducing the complexity ofimage processing.

According to an embodiment of the present disclosure, the fourth weightvalue corresponding to the first image region tends to decrease from thecenter axis to the edges on both sides, and the fifth weight valuecorresponding to the second image region tends to decrease from thecenter point to the edges of the first region of interest. By processingthe image in combination with the subsequent steps, it is possible tomake the image change smoother and avoid the sudden transition betweenthe region of interest and the region of non-interest, thereby improvingthe display effect of the image. According to the image processingmethod provided by the embodiment of the present disclosure, theprocessed image can maintain the original state of the image in thefirst region of interest, and enable the chroma and contrast of theimage in the regions other than the first region of interest to benaturally transitioned according to the weight values of the pixelpoints, therefore making the display effect of the image better.

After the weight value of each pixel point in the to-be-processed imageis determined, it proceeds to step S204.

S204: adjusting the pixel value of each pixel according to the weightvalue corresponding to each pixel so as to acquire the processed image.

In one embodiment, the step of adjusting the pixel value of each pixelaccording to the weight value corresponding to each pixel so as toacquire the processed image may further include S701-S702.

S701: determining image information Qstep corresponding to each pixel inthe to-be-processed image according to a pre-established mappingcorrespondence between the weight value and an image information Qstep.

The image information Qstep may be used to re-quantize the pixel valuesof the pixel points. The image information Qstep may be set according tothe representation format of the pixel dots. For example, for the RGB 24format, that is, each pixel point represents an RGB value (pixel value)with 24 bits, the value of the image information Qstep ranges from 1 to256, and the image information Qstep may be 1, 2, 32, or 64, forexample. Further, different image information Qstep is set for differentweight values in advance, and the mapping correspondence between theweight value corresponding to the pixel point and the image informationQstep is stored. For example, a smaller image information Qstep is setfor a higher weight value, and a larger image information Qstep is setfor a lower weight value.

S702: adjusting the pixel value of each pixel according to the imageinformation Qstep corresponding thereto so as to acquire the processedimage.

The process of adjusting the pixel value of each pixel according to theimage information Qstep corresponding thereto is described below with aspecific example.

Taking a 24-bit pixel point with 3 colors as an example, it is assumedthat the pixel value of the pixel point X in the to-be-processed imageis (R=117, G=63, B=244), the weight value corresponding to the pixelpoint X is relatively low, and the image information Qstep correspondingto the pixel points X is 64. The R component of 117 is in the intervalof 64-128, and the median of 96 of the interval is taken as there-quantized R component; the G component of 63 is in the interval of0-64, and the median of 32 of the interval is taken as the re-quantizedG component; and the B component of 244 is in the range of 192-256, andthe median of 224 of the interval is taken as a re-quantized Bcomponent; that is, the adjusted pixel value (RGB value) of the pixelpoint X is (R=96, G=32, B=224).

It is assumed that the pixel value of the pixel point Y in theto-be-processed image is (R=81, G=223, B=129), the weight valuecorresponding to the pixel point Y is relatively low, and the imageinformation Qstep corresponding to the pixel points Y is 2. The Rcomponent of 81 is in the interval of 80-82, and the median of 81 of theinterval is taken as the re-quantized R component; the G component of223 is in the interval of 222 224, and the median of 223 of the intervalis taken as the re-quantized G component; and the B component of 129 isin the range of 128-130, and the median of 129 of the interval is takenas a re-quantized B component; that is, the adjusted pixel value (RGBvalue) of the pixel point X is maintained as (R=81, G=223, B=129).

According to the embodiment of the present disclosure, by associatingthe weight value corresponding to a pixel point with the imageinformation Qstep, for a pixel point with a lower weight value, a largerimage information Qstep may be used to re-determine the pixel value, sothat it is possible to reduce the image information of the pixel point,so as to reduce the complexity of image processing and reasonably reducethe chroma and contrast of the local area of the to-be-processed image;for a pixel point with a higher weight value, a smaller imageinformation Qstep may be used to re-determine the pixel value, so thatmore image information may be retained to ensure the quality of theimage.

According to the embodiment of the present disclosure, for the pixelpoints located in the same image region, the pixel values of the pixelpoints are adjusted by using the same image information Qstep, so thatit is possible to reduce the complexity of image processing.

In one embodiment, after the processed images are acquired, the imageprocessing method further includes: performing video coding on thesequentially arranged multi-frame processed images to acquire the targetvideo.

According to that embodiment of the present disclosure, the first regionof interest of the eye on the to-be-processed image is predict accordingto the eyeball-tracking information and the weight value is assigned toeach pixel point in the to-be-processed image. By adjusting the pixelvalue of the pixel point according to the weight value corresponding tothe pixel point, it is possible to reduce the complexity of imageprocessing while ensuring the image quality, thereby possibly reducingthe complexity of video coding and improving the speed of video coding.In addition, it is possible to reduce the volume of the encoded video,improve the transmission speed of the video, reduce the display delay,and improve the user experience.

In one embodiment, after the processed image is acquired, the imageprocessing method further includes S801-S803.

S801: performing convolution operation on the processed image to obtaina convolution operation result.

S802: determining the second region of interest of the processed imageaccording to the convolution operation result.

In this embodiment, for an image with a width of w and a height of h, aconvolution kernel with a width of w/4 and a height of h/4 may be usedto traverse the processed image, thereby acquiring a convolutionoperation result for each convolution kernel. The convolution operationresult may be the sum of weight values of the convolution kernel, andthe region with the highest sum of weight values of the convolutionkernel is counted, which is the second region of interest.

S803: carrying out video coding on the plurality of frames of processedimages arranged in sequence with a predetermined region-of-interestcoding algorithm based on the second region of interest so as to obtaina target video.

The predetermined region-of-interest coding algorithm may be, forexample, the h264 algorithm or the h265 algorithm.

According to the embodiment of the present disclosure, by carrying outvideo coding on the plurality of frames of processed images arranged insequence with a predetermined region-of-interest coding algorithm basedon the second region of interest, it is possible to enhance region ofinterest. According to the embodiment of the present disclosure, byperforming convolution operation on the processed image, it is possibleto make the image smoother.

Apparatus Embodiment 1

Referring to FIG. 7 , an embodiment of the present disclosure providesan image processing apparatus 70 for a head-mounted display device, andthe image processing apparatus 70 includes an acquisition module 71, afirst region-of-interest determining module 72, a weight valuedetermining module 73, and an image processing module 74.

The acquisition module 71 is configured to acquire the to-be-processedimage and the eyeball-tracking information.

The first region-of-interest determining module 72 is configured todetermine a first region of interest of the eyeball on theto-be-processed image based on the eyeball-tracking information.

In one embodiment, the eyeball-tracking information includes the centerpoint and radius of the fixation area of the eyeball on theto-be-processed image, and the maximum movement angle of eyeball on theto-be-processed image within the reception interval.

The first region-of-interest determining module 72 is specificallyconfigured to determine the maximum displacement of the eyeball on theto-be-processed image within the reception interval according to themaximum movement angle.

The first region-of-interest determining module 72 is specificallyfurther configured to determine the first region of interest accordingto the center point of the fixation area and the maximum displacement ifthe maximum displacement is greater than or equal to the radius of thefixation area.

The first region-of-interest determining module 72 is specificallyfurther configured to determine the fixation area as the first region ofinterest if the maximum displacement is lower than the radius of thefixation area.

In one embodiment, the image processing device 70 further includes amaximum movement angle determining module.

The maximum movement angle determining module is configured to obtain anactual movement angle of the eyeball on the to-be-processed image withinthe reception interval.

The maximum movement angle determining module is further configured totake the field of view of the head-mounted display device as the maximummovement angle of the eyeball on the to-be-processed image within thereception interval if the actual movement angle is greater than or equalto the field of view of the head-mounted display device.

The maximum movement angle determining module is further configured totake the actual movement angle as the maximum movement angle of theeyeball on the to-be-processed image within the reception interval ifthe actual movement angle is lower than the field of view of thehead-mounted display device.

The weight value determining module 73 is configured to assign a weightvalue to each pixel in the to-be-processed image, wherein, the weightvalue of the pixel point located in the first region of interest ishigher than the weight value of the pixel points located in otherregions, and the other regions are regions other than the first regionof interest.

In one embodiment, the weight value determining module 73 isspecifically configured to assign a first weight value to each pixel inthe to-be-processed image, wherein the first weight value correspondingto the pixel point tends to decrease from the central axis to the edgeson both sides.

The weight value determining module 73 is specifically configured toassign a second weight value to each pixel point in the first region ofinterest, and the second weight value corresponding to the pixel pointtends to decrease from the center to the edge of the first region ofinterest.

The weight value determining module 73 is specifically configured toperform weighted average calculation on the first weight value and thesecond weight value corresponding thereto in the first region ofinterest, so as to determine the third weight value of each pixel pointin the first region of interest.

In one embodiment, the weight value determining module 73 is furtherspecifically configured to perform partition on the to-be-processedimage, so as to obtain a plurality of first image regions.

The weight value determining module 73 is further specificallyconfigured to assign a fourth weight value to each of the first imageregions, wherein the fourth weight value corresponding to the firstimage region tends to decrease from the central axis to the edges onboth sides.

The weight value determining module 73 is further specificallyconfigured to perform partition on the first region of interest, so asto obtain a plurality of second image regions.

The weight value determining module 73 is further specificallyconfigured to assign a fifth weight value to each second image region,wherein the fifth weight value corresponding to the second image regiontends to decrease from the center point to the edges of the first regionof interest.

The weight value determining module 73 is further specificallyconfigured to determine the weight value of each pixel point in theto-be-processed image according to the fourth weight value correspondingto the first image region where the pixel point is located and the fifthweight value corresponding to the second image region where the pixelpoint is located.

The image processing module 74 is configured to adjust the pixel valueof each pixel point according to the weight value corresponding thereto,so as to obtain the processed image.

In one embodiment, the image processing module 74 is specificallyconfigured to determine image information Qstep corresponding to eachpixel point in the to-be-processed image according to a pre-establishedmapping correspondence between the weight value and the imageinformation Qstep.

The image processing module 74 is further configured to adjust the pixelvalue of each pixel point according to the image information Qstepcorresponding thereto, so as to obtain the processed image.

In one embodiment, the image processing device 70 further includes avideo coding module 75.

The video coding module 75 is configured to perform video coding on aplurality of frames of the processed images arranged in sequence, so asto obtain the target video.

In one embodiment, the video coding module 75 is further configured toperform a convolution operation on the processed image, so as to obtainthe convolution operation result.

The video coding module 75 is further configured to determine the secondregion of interest of the processed image according to the convolutionoperation result.

The video coding module 75 is further configured to, based on the secondregion of interest, perform video coding on the plurality of frames ofthe processed images arranged in sequence with the predeterminedregion-of-interest coding algorithm, so as to obtain the target video.

Referring to FIG. 8 , an embodiment of the present disclosure providesan image processing apparatus 80 for the head-mounted display device,and the image processing apparatus 80 includes a processor 81 and amemory 82. The memory 82 is configured to store a computer program that,when run by the processor 81, implements the image processing method forthe head-mounted display device as disclosed in any of the precedingembodiments.

Apparatus Embodiment 2

An embodiment of the invention also provides an electronic device. Theelectronic device includes a head-mounted display device and an imageprocessing apparatus.

In one embodiment, the head-mounted display device may be thehead-mounted display device 1000 as shown in FIG. 1 . In one embodiment,the head-mounted display device may be smart devices such as a virtualreality (VR) device, an augmented reality (AR) device or a mixed reality(MR) device.

In one embodiment, the image processing apparatus may be the imageprocessing apparatus 70 for the head-mounted display device as shown inFIG. 7 , or the image processing apparatus 80 for the head-mounteddisplay device as shown in FIG. 8 . In one embodiment, the imageprocessing apparatus may be a PC end 2000 as shown in FIG. 1 .

The image processing apparatus is connected to the head-mounted displaydevice. The image processing apparatus is configured to process theto-be-processed image according to the pose data of the head-mounteddisplay device, perform video coding on the plurality of frames of theprocessed images arranged in sequence to obtain the target video, andtransmit that target video to the head-mounted display device. Thehead-mounted display device is configured to present the target video tothe user.

According to the embodiment of the present disclosure, the first regionof interest of the eyeball on the to-be-processed image is predictedaccording to the eyeball-tracking information, each pixel point in theto-be-processed image is assigned a weight value, and the pixel value ofthe pixel point is adjusted according to the weight value correspondingto the pixel point, so that it is possible to reduce the complexity ofimage processing while ensuring the image quality.

According to the embodiment of the present disclosure, the processedimage can maintain the original state of the image in the first regionof interest, and enable the chroma, contrast, and the like of the imageto be naturally transitioned according to the weight values of pixelpoints in regions other than the first region of interest, thereforemaking the display effect of the image better.

Each embodiment in this specification is described in a progressivemanner, and the same or similar parts between the various embodimentsmay refer to each other. Each embodiment focuses on the differences fromother embodiments, but it should be clear to those skilled in the artthat the above embodiments may be used individually or in combinationwith each other as required. In addition, the description of the deviceembodiments is relatively simple, since they correspond to the methodembodiments, and therefore, the relevant part may refer to thedescription of the corresponding part of the method embodiment. Thesystem embodiments described above are merely illustrative, wherein themodules described as separate components may or may not be physicallyseparate.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium may be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein may bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may includecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, may be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein includes anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which includes one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, may be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions. It is well-known toa person skilled in the art that the implementations of using hardware,using software or using the combination of software and hardware may beequivalent with each other.

The descriptions of the various embodiments of the present inventionhave been presented for purposes of illustration, but are not intendedto be exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the describedembodiments. The terminology used herein was chosen to best explain theprinciples of the embodiments, the practical application or technicalimprovement over technologies found in the marketplace, or to enableothers of ordinary skill in the art to understand the embodimentsdisclosed herein. The scope of the present invention is defined by theattached claims.

1. An image processing method for a head-mounted display device,comprising: acquiring a to-be-processed image and eyeball-trackinginformation; determining a first region of interest of an eyeball on theto-be-processed image according to the eyeball-tracking information;assigning a weight value to each pixel point in the to-be-processedimage, wherein weight values of pixel points located in the first regionof interest are higher than weight values of pixel points located inother regions which are regions other than the first region of interest;adjusting a pixel value of each pixel point according to the weightvalue corresponding thereto so as to obtain a processed image.
 2. Themethod of claim 1, wherein the eyeball-tracking information comprises acenter point and a radius of a fixation area of the eyeball on theto-be-processed image, and the maximum movement angle of the eyeball onthe to-be-processed image within a reception interval; determining afirst region of interest of an eyeball on the to-be-processed imageaccording to the eyeball-tracking information comprises: determining themaximum displacement of the eyeball on the to-be-processed image withinthe reception interval according to the maximum movement angle;determining the first region of interest according to the center pointand the maximum displacement of the fixation area if the maximumdisplacement is larger than or equal to the radius of the fixation area;determining the fixation area as the first region of interest if themaximum displacement is smaller than the radius of the fixation area. 3.The method of claim 2, wherein the method further comprises determiningthe maximum movement angle of the eyeball on the to-be-processed image,comprising: acquiring an actual movement angle of the eyeball on theto-be-processed image within the reception interval; taking a field ofview of the head-mounted display device as the maximum movement angle ofthe eyeball on the to-be-processed image within the reception intervalif the actual movement angle is larger than or equal to the field ofview of the head-mounted display device; taking the actual movementangle as the maximum movement angle of the eyeball on theto-be-processed image within the reception interval if the actualmovement angle is smaller than the field of view of the head-mounteddisplay device.
 4. The method of claim 1, wherein assigning a weightvalue to each pixel point in the to-be-processed image comprises:assigning a first weight value to each pixel point in theto-be-processed image, wherein the first weight value corresponding tothe pixel point tends to decrease from a central axis of theto-be-processed image to edges on both sides of the same; assigning asecond weight value to each pixel point in the first region of interest,wherein the second weight value corresponding to the pixel point tendsto decrease from a center to edge of the first region of interest;performing weighted average calculation on the first weight value andthe second weight value corresponding thereto in the first region ofinterest, so as to determine a third weight value of each pixel point inthe first region of interest.
 5. The method of claim 1, whereinassigning a weight value to each pixel point in the to-be-processedimage comprises: partitioning the to-be-processed image so as to obtaina plurality of first image regions; assigning a fourth weight value toeach of the first image regions, wherein the fourth weight valuecorresponding to the first image regions tends to decrease from acentral axis of the to-be-processed image to edges on both sides of thesame; partitioning the first region of interest so as to obtain aplurality of second image regions; assigning a fifth weight value toeach of the second image regions, wherein the fifth weight valuecorresponding to the second image regions tends to decrease from acenter point of the first region of interest to an edge of the same;determining a weight value of each pixel point in the to-be-processedimage according to the fourth weight value corresponding to the firstimage region where the pixel point is located and the fifth weight valuecorresponding to the second image region where the pixel point islocated.
 6. The method of claim 1, wherein adjusting a pixel value ofeach pixel point according to the weight value corresponding thereto soas to obtain a processed image comprises: determining an imageinformation Qstep corresponding to each pixel point in theto-be-processed image according to a pre-established mappingcorrespondence between the weight value and the image information Qstep;adjusting the pixel value of each pixel point according to the imageinformation Qstep corresponding thereto so as to obtain a processedimage.
 7. The method of claim 1, wherein the method further comprises:performing video coding on a plurality of frames of the processed imagearranged in sequence so as to obtain a target video.
 8. The method ofclaim 1, further comprising: performing convolution operation on theprocessed image so as to obtain a convolution operation result;determining a second region of interest of the processed image accordingto the convolution operation result; based on the second region ofinterest, performing video coding on a plurality of frames of theprocessed image arranged in sequence with a predeterminedregion-of-interest coding algorithm so as to obtain a target video. 9.An image processing apparatus for a head-mounted display device,comprising: an acquisition module configured to acquire ato-be-processed image and eyeball-tracking information; a firstregion-of-interest determining module configured to determine a firstregion of interest of an eyeball on the to-be-processed image accordingto the eyeball-tracking information; a weight value determining moduleconfigured to assign a weight value to each pixel point in theto-be-processed image, wherein weight values of pixel points located inthe first region of interest are higher than weight values of pixelpoints located in other regions which are regions other than the firstregion of interest; an image processing module configured to adjust apixel value of each pixel point according to the weight valuecorresponding thereto so as to obtain a processed image; or, theapparatus comprises a processor and a memory, the memory storingcomputer instructions that, when run by the processor, execute themethod of claim
 1. 10. An electronic device, comprising a head-mounteddisplay device and an image processing apparatus for the head-mounteddisplay device of claim 9.